Voice over Internet Protocol (VoIP) in a Control Center 

Environment 


Joseph L. Pirani 1 

NASA Marshall Space Flight Center, Huntsville, Alabama, 35812 
Steven G. Calvelage 2 

Co Is a Corporation, Marshall Space Flight Center, Huntsville, Alabama, 35812 


The technology of transmitting voice over data networks has been available for over 10 
years. Mass market VoIP services for consumers to make and receive standard telephone 
calls over broadband Internet networks have grown in the last 5 years. While operational 
costs are less with VoIP implementations as opposed to time division multiplexing (TDM) 
based voice switches, is it still advantageous to convert a mission control center’s voice 
system to this newer technology? Marshall Space Flight Center (MSFC) Huntsville 
Operations Support Center (HOSC) has converted its mission voice services to a commercial 
product that utilizes VoIP technology. Results from this testing, design, and installation have 
shown unique considerations that must be addressed before user operations. There are many 
factors to consider for a control center voice design. Technology advantages and 
disadvantages were investigated as they refer to cost There were integration concerns which 
could lead to complex failure scenarios but simpler integration for the mission 
infrastructure. MSFC HOSC will benefit from this voice conversion with less product 
replacement cost, less operations cost and a more integrated mission services environment. 


I. Introduction 

V OICE over Internet Protocol (VoIP) has changed the world of telephony. Businesses are now phasing out 
traditional phone lines in favor of their data network lines. With new construction or remodeling, it is less 
expensive to just pull data cable, particularly Category 5 cable, to run voice and data in all locations in a 
building. Businesses with multiple locations can consolidate phone services with Internet telephony for less cost 
than traditional phone service at each location. Residential VoIP services have grown with more broadband 
availability throughout the country. 

There are advantages and disadvantages of using VoIP services. The main advantage is the cost. One 
aspect of that cost is the network infrastructure. The same network that carries data also carries voice traffic. That 
equates to less wiring in new building installations or remodeling and less electronic infrastructure because of the 
converged service. Because residential VoIP services are not taxed or assessed regulatory fees, the service is much 
less expensive than traditional phone service. PC to PC calls are usually free. Another advantage is portability. 
With a laptop, you can always log into your VoIP account and make or receive telephone calls with the same 
assigned telephone number. This could be an advantage over the portability of cell phones if you are in a different 
country as there would be no new fees for ‘roaming’ or connecting to another subscriber cell network. This 
portability also allows businesses to consolidate their multiple locations with a single phone network. It also works 
in the reverse example; a business can have a virtual presence in another geographic location. Your business can 
have a Houston, Texas phone number, but the number will connect to a phone in Huntsville, Alabama. 

There are also disadvantages to VoIP services. The main disadvantage is service during a power outage. 
Plain old telephone service (POTS) phones are powered through the current supplied through the phone line. Your 
VoIP equipment is locally powered and susceptible to an interruption in power. The business must also add the cost 
of a generator or uninterruptible power supply to the VoIP service if it expects voice service during power outages. 
While portability is an advantage, it also leads to a disadvantage of emergency calls. Traditional 911 services can 
trace the location of an emergency call and divert to the nearest call center. VoIP calls are made between IP 
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addresses and not physical locations. The technology is not there to pinpoint the actual location of an IP address 
where a VoIP call originates. One of the earlier disadvantages of sharing voice and data traffic on the Internet was 
voice quality. During large data transfers, voice service would suffer because all IP traffic is the same. There was 
no way to prioritize which data packets were more important for timely delivery. Voice quality has improved and 
major networking vendors have introduced equipment to route and prioritize voice traffic. VoIP services and 
installations have increased in the last 5 years. However, residential broadband Internet service though Digital 
Subscriber Line (DSL) or television cable providers is still not as reliable as the local telephone company. So, 
customers are still weighing the cost versus availability options in deciding to switch from POTS to VoIP. So, this 
is one of many choices that an Information Technology (IT) organization must make when providing voice services 
to a business or campus. 


IT. History of VoIP at NASA Marshall Space Flight Center 

In 2002, Marshall Space Flight Center (MSFC) had to replace their Private Area Branch Exchange (PABX) 
phone system. This system had been in operation since the mid 1980’s. VoIP was chosen as a means to replace 
standard telephone service in a selected area. This was needed to gather existing phone switch hardware for spare 
parts. As the PABX switch was no longer being supported by the vendor, it was imperative that enough parts could 
be gathered for maintenance and repairs until such time that a new replacement system could be ordered and 
installed. 

VoIP was a viable commercial option at the time. During this timeframe, MSFC was upgrading its 
networking infrastructure to category 5 cabling and 100BASE-TX Ethernet standards. One area of buildings on the 
campus was undergoing facility renovations. Users in these buildings were selected to use the new VoIP technology 
to test the service and user reactions to the new phones. This experiment concluded at the end of the fiscal year 
2002. Many of the disadvantages of VoIP mentioned earlier had real effects on the project. In an attempt to make 
the new voice services as close to reliable as the current PABX system, it was imperative to increase the network 
reliability. Redundancy was designed into the network. Core and distribution routers were redundantly installed to 
handle a failure of any one router. However, this redundancy was not installed at the end user switches where user 
devices, such as personal computers or VoIP telephones, were connected. Uninterruptible Power Supplies (UPS) 
were priced to ensure no power interruptions to phone service. Core and distribution routers were connected to UPS 
power. End user switches were not connected to UPS power. The network vendor had newer switches that could 
provide line power to a network attached device. These switches were significantly more expensive. Also, VoIP 
phones were much more expensive than digital telephone devices. 

Extrapolating the costs of adding reliable power to a VoIP system with the cost of new phone units across 
the entire campus, VoIP services in 2002 would have been twice as much cost as simply buying another PABX. So, 
the VoIP experiment ended at MSFC for normal telephone service. 

in. Voice Services at the Huntsville Operations Support Center (HOSC) 

The HOSC, located at the Marshall Space Flight Center, supports the International Space Station (ISS) 
Payload Operations Integration Center (POIC). The POIC manages the execution of on-orbit ISS payloads and 
payload support systems in coordination with the Mission Control Center in Houston (MCC-H), the distributed 
International Partner Payload Control Centers, Tele-science Support Centers, and payload-unique facilities 
throughout the world. The POIC provides capabilities required by the POIC Cadre to manage and integrate payload 
operations. The POIC also provides capabilities required by individual payload users to operate and control their 
payloads and experiments. In addition to telemetry and command processing and video distribution, the POIC 
provides mission voice communication. The POIC is the source of payload voice communications to all payload 
users. 

The circuit-switched HOSC voice system, the Enhanced Voice Distribution System (EVoDS), was installed 
in 1999 utilizing existing keysets that had been in use since the late 1980’s. The system provided standard voice 
conferencing services via HOSC operator configuration control of keysets. It also provides record and playback 
capability for internal HOSC users. Remote facilities and users have the capability to connect to the HOSC via T1 
interface. The system does provide standard Air-to-Ground and Space to Ground voice loop enable and disable 
capability for remote sites and keysets. 

In 2000, the Missions Operations Lab (MOL) embarked on a joint development with industry to provide an 
Internet based voice distribution capability for the mission science community. During this timeframe, the HOSC 
began delivering more remote services. The paradigm had shifted from payload operations during Shuttle missions 
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with a large contingent of scientists and payload operators located in the HOSC to 24-hour, on-orbit science with 
scientists and payload operators located anywhere. It was impractical for these users to relocate to Huntsville for 
mission increments which could last six months or more. In May 2002, the MOL released an Internet Voice 
Distribution System (IVoDS) for remote payloads users. In addition to VoIP conferencing servers, connections to 
the HOSC circuit-switched voice system, the IVoDS architecture also included a virtual private network (VPN) 
connection to the HOSC as this voice traffic traversed the general Internet. Figure 1 shows the general voice 
architecture of the HOSC as VoIP services were added in 2002. 
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Figure 1 : HOSC Voice Architecture circa 2002 1 


Although the cost for such a development was relatively high, the ISS program realized that the savings of utilizing 
this technology versus the travel costs plus installing dedicated voice circuits to remote locations was well worth the 
initial cost. Also, the advantages of this cutting edge technology outweigh the disadvantages of potential quality of 
service issues with the Internet. A comprehensive testing program, which lasted over six months and involved 
schools and NASA National Educators Workshop teachers, provided feedback and the initial baseline for the voice 
quality when the service was introduced to ISS operations. 2 

In 2007, research showed that VoIP technology had progressed to allow for a commercial-off-the-shelf 
(COTS) system. A COTS replacement system was purchased and installed for POIC remote users. This latest 
technology opened the doors for allowing voice to practically anywhere via a laptop with Internet connectivity. 
Another paradigm shift was occurring as research was beginning to prove that using a total IP voice distribution 
switch was a viable option for a control center environment. However, the EVODS has gone well beyond end of 
service agreements with its vendor. Also, VoIP had improved as a commercial product since 2002, and many users 
wanted better performance and features similar to that of a traditional voice switch keyset. It was past time for the 
HOSC to upgrade its voice services for the POIC. 

In 2007, NASA finalized an Agency purchase for mission voice switches. The Mission Operations Voice 
Enhancement (MOVE) would replace existing mission voice systems with COTS products to standardize these 
systems throughout NASA. However, some of HOSC’s requirements were not met with MOVE, particularly the IT 
security requirements, as the “one size fits all” philosophy could not keep up with the evolving mission support 
philosophy of the POIC. As mentioned earlier, the POIC supports many remote users. The paradigm shift of 
delivering services was no longer confined to users in a control room environment. As the MOVE system was time 
division multiplexed (TDM) based and common to all Centers being delivered a new system, implementing new 
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requirements could take years. A server based IP voice system was more flexible and more responsive to the 
POIC’s changing requirements and user base. The MOVE system is a purely TDM solution and contractually in 
place for 10 years. Since it meets NASA rigid static configurations required for Mission Critical Real time manned 
space flight requirements, it does not necessarily facilitate less static environments such as space science research 
and payload management. The HOSC had to adapt its mission voice communications services to IP based voice 
solutions to meet the constantly changing customer base. 

IV. The Future of VoIP at the HOSC 

The Voice over IP Mission Conferencing environment considers many of the same support requirements as 
would be for a telephony design and implementation. First, the HOSC team had to determine how large this 
Mission Conferencing VoIP system should be to handle the ISS mission at MSFC. The POIC supports 100 
concurrent users accessing up to 500 unique conferences, with over 700 external participants joining in the form of 
T1 DS0 digital signaling connections. Using original EVoDS telephony requirements and enhancing these 
requirements with existing remote IVoDS requirements, Table 1 lists a blend of telephony and IT requirements for 
the system. 


950 real time conferences 

Access control to restricted conferences 

Individual user ID’s 

Encryption and hardened information 
technology security (ITS), with account 
lock out 

Individual user preferences 

Lightweight Directory Access Protocol 
(LDAP) Interoperable 

Multiple headsets, with external Speakers and 
Foot Switches 

Multiple compressor- decompressor 
(CODEC’s) support 

Sidetone 

Gain and attenuation controls 

Non blocking 

Rapid reconfiguration 

Special signaling controls and support 

System alarm and status 

External interfacing via a channelized T1 



Table 1: Mission Voice Conferencing IVoDS Requirements. This listing is not all inclusive 
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Unlike a traditional telephone system, telephony included, the Mission Voice Conferencing systems allow users 
access to hundreds of conferences at the push of a button: no dialing, no call set up, no waiting. Specification for 
transcoding dissimilar streams and media termination point bridging were not included for the Mission Voice 
Conferencing system versus being mandatory features of a telephony system. 

A major design consideration for either a telephony or Mission Voice Conferencing system is the 

underlying network infrastructure. The 
POIC determined early on that the 
Mission Voice services warranted 
dedicated switches and LAN segments 
to the core routing engine to connect the 
user devices, and the conferencing and 
gateway servers. This decision 
eliminates the risk of traffic competition 
for available bandwidth and priorities. 
This is not a luxury for deploying a 
telephony solution, which relies heavily 
on the use of existing infrastructure to 
be cost beneficial. Figure 2 shows the 
IVoDS system architecture. As 
mentioned earlier, a disadvantage of 
VoIP is the quality of voice could suffer 
if IP traffic has suddenly increased on 
the data network. Mitigation to this risk 
is to implement a set of products and 
techniques which would prioritize voice 
traffic above other data. This is an 
Figure 2: IVoDS System Architecture 2010 added cost to network equipment and is 

labor intensive. An additional risk is having all traffic at risk during a network outage. HOSC mitigated this risk by 
having a separate local network (LAN) for voice traffic within the control center. The cost was minimal to add this 
separate LAN as opposed to upgrading the existing network to prioritize traffic and to eliminate additional potential 
failure points in the network. 

Voice services and systems have to be cognizant of the end to end time as to minimize the effects of delay 
that can be perceived as echo. A range of 25 to 150 milliseconds (ms) is generally considered an acceptable range. 
This is the time measured from speech source to listener ear. The distance between any two users will always 
involve many different types of equipment, of analog and digital communications functions. This is can be 
described as; (Talker) User Headset (H/S) to Conference/Router to Gateway to LAN/wide area network (WAN) to 
Gateway to Switch to User H/S (Listener). Since every user hears and talks in analog, this sound energy has to be 
converted to a digital form at each end preferably with no other conversion in between. This conversion should take 
place in very close proximity to the actual ear and mouth. The digital conversion is described as Mu Law per the 
ITUG.711 3 standard in which voice is sampled 8000 times per second with each sample representing 8 bits, which 
yields the highest speech quality. Once digitized, the voice packets conform to standard digital communications 
architectures. 

The challenge for a Control Center and its voice users is that generally the spoken conversations are from 
one to many versus a point to point connection. A portion of those many are located in the same proximity or room, 
while others could be located thousands of miles away. Prudent network planning would be used in the Control 
Center design and implemented to limit the delay between all users to < 150ms 4 . Legacy voice conferencing 
systems, which are usually TDM switches, overcome this challenge through the use of dedicated wiring and circuits 
that operate at high speeds. This works well but does not allow itself to collaborative networking. Dedicated TDM 
systems generally meet or exceed delay times <80ms including any terrestrial WAN links. In an all or near all IP 
system, new considerations are mandatory such as the Open Systems Interconnection model for layered 
communications and computer networking as well as other networking protocols such as TCP/IP, UDP, and 
Signaling Standards. Special considerations are given to the general make up of co-located and inter-room listeners, 
where visual contact as well as ambient sounds can be observed. The goal in designing an IP voice system to meet 
a Control Room environment is to be standards based and to have minimal software customization to replicate 
legacy time division multiplexed functions. 
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A commercial system was implemented in early 2010 within the 
POIC. The newest IVoDS service has minimal levied developmental 
requirements to meet the delay expectation of 60 to 120ms. Comparing 
the differences between the standard HOSC IVoDS service that provides 
voice services to remote science community, and the latest delay 
intolerant IVoDS, the round trip delays are quite contrasting. One way 
packet delivery dropped from 210ms to 120ms. Expecting delays of 
120ms is worst case and it generally averages around 100ms. The biggest 
offender of delay is induced by the Operating System (OS) and 
application software routines. The network (layer 3 and down) only plays 
a small role of a few milliseconds. This is due to implementing a network 
that is 100Base-T from network interface card to switch and gigabit over 
fiber between the core routers. As described earlier, traffic is segmented 
to its own network and dedicated switches. The analog to digital 
conversion takes place in a commercial universal serial bus (USB) attached sound card in a traditional pistol grip 
style with integrated push-to-talk (PTT) function. Another option to achieve reductions in voice processing 
handling is in the form of developing a custom digital signal processor and network interface that performs a very 
specific function. This type of effort would utilize an existing user PC to connect the add-on custom hardware. No 
activities were performed to test the custom digital serial processor idea that the voice processing was expected to 
perform around the 80ms range. 

The user keysets, as shown in Figure 3, have been developed using a light weight Unix OS running on a 
compact dual core, single board computer package. Many of the usual OS functions have been removed and 
tailored to focus on digital voice processing with some input-output management to operate a Touch Screen device 
for user interactions. Pursuing the customized OS provided another equally important benefit. By removing most of 
the services running on a standard OS, the IVoDS Keysets proved very secure. By reporting few, if any, 
vulnerabilities on all IT security scans, the keyset is an invisible (non reactive) device hanging on the network. It is 
truly an appliance running on the network that only requires two ports for secure encrypted voice communications 
between any user and the conferencing server. 

With the high security provided by the new IVoDS keysets, the system integrates with all current network 
and institutional services, such as Active Directory, network timing protocol (NTP), Enterprise network operations 
center (NOC) monitoring and control, voice recorders and external voice lines and connectivity. Additionally, it is 
well suited to integrate with future requirements including single sign-on and two factor authentication. The 
forward looking plan for the Mission Voice over IP services is a candidate for the contingency operations and 
mobile computing environments. 
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Figure 4: HOSC IVoDS User Interface 
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V. Operations Paradigm Shift 

An evaluation of operational support skills was done prior to selecting the new local IVoDS service into the 
POIC. Traditional TDM telephone systems require specialized tools and technical skills to maintain and operate. 
The new system brings established skills that currently reside in the facility, such as network engineers, systems 
management and administration, account management and IT security. Contrasting the skills above with a TDM 
system, these are not normally integrated with many of the standard services, and requires talent not seen much 
today such as the telecommunications engineer, electronics technician and field electronics technicians. Skills to 
troubleshoot electronics circuits to component or sub component level have been replaced by systems that are either 
replaced in whole or broken down to assemblies like line replaceable units; hard disk drives, power supplies, 
motherboards, memory modules and Peripheral Component Interconnect (PCI) cards. The new IVoDS is no 
different in terms of maintenance as to what would be expected to operate and maintain a personal computer or 
server. These are the skills sets of today and tomorrow. Similarly, the users are expected to operate the new user 
devices, which resemble any other computer or workstation located on the POIC consoles. Figure 4 shows the 
IVoDS user interface. From an operations day to day management perspective, it is truly a point and click 
environment, compared to the old days of single command style line entry. 

The new system was implemented in parallel with the existing system for some length of time, in order for 
systems validation, and training to occur. Particularly high on the list of items to accomplish was a short parallel 
real time operations period. This is driven by the need for a user acclimation period enabled both the end user and 
support personnel to acquire the knowledge base to operate the new system proficiently. There was also a concern 
of induced human errors while maintaining multiple voice systems, as skill sets were different for each system. A 
minor facility concern consisted of sustained burdens on the facility as the legacy system power and cooling 
consumption levels were near the maximum safe parameters. The older TDM systems consume large amounts of 
power and require much more cooling than the new IVoDS. A short 2-3 month period was targeted for the POIC 
parallel operations. 

With the new system in place, it is clear that operations and maintenance will not add additional complexity 
to the Ground Operations model. As shown, the new voice system simplifies the operations model by replacing a 
unique TDM voice system with an IP voice system that has the same IT footprint as other control center servers. 
The new system provides the mission voice users with a flexible yet highly available service. IVoDS now supports 
the internal and external customers of the POIC and HOSC. 

HOSC collaborated with the Johnson Space Center (JSC) Mission Operations Directorate (MOD) for many 
projects to find common solutions to both centers. As part of that collaboration, the POIC has tried to lead in 
implementing more cutting edge solutions to provide a proof of concept for NASA’s main mission control center at 
JSC. MOD is also getting more requests for mission voice loops outside of their control center environment. The 
HOSC is showing the benefits and difficulties of remote servicing of voice services to MOD. Their implementation 
should be easier and with less risk, benefiting from the experiences of implementing VoIP services at the HOSC. 

VI. Conclusion 

MSFC HOSC was in desperate need to replace its mission voice system. The EVODS system was well 
beyond its end of serviceable life. The HOSC has annual power maintenance on its electrical grid. Coming out of 
this service window each year, it was always a risk whether EVODS would survive the power cycle. Now, with a 
new system with more modem technology and a smaller footprint, the HOSC will start to see operations and 
maintenance savings. The first immediate savings will be space. The new IVoDS system resides in the same 
computer room as the rest of the control center servers. EVODS, with its many racks, interfaces and wiring 
infrastructure, along with residing in its own room is a methodology of the past. This will be, obviously, less draw 
on the total facility power consumption steering the HOSC towards a greener business model. 

The users quickly picked up on the use and manipulation of the keysets to customize it to their preference. 
Some of these user specific settings included loop location on the key frame matrix, individual volume control, 
screen angle, brightness and color saturation. A survey was given to each user after group and individual training on 
the new system. Table 2 summarizes the overall results from the collected data concerning three targeted areas of 
the service and keysets. Obviously, the new keysets are a paradigm shift from the push button keysets from over 
twenty years ago. Each user now controls customization features of the keyset to allow him to operate as he feels 
most efficient. These customized configurations are saved for each user as the keyset and system is accessed 
through individual login. 

Voice over IP introduces a new chapter in mission operations. Service costs are reduced and users have 
more flexibility over their voice services. With appropriate permissions, users outside of the control center can now 


7 

American Institute of Aeronautics and Astronautics 


participate in mission operations as society has become more dependent on mobility and flexibility of 
communications. 



U*er Operability 
Keyset U^&abilkv 
Audio Quality 


Table 2: IVoDS User Survey Results 
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